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IMAGE PROCESSING METHOD AND APPARATUS 

BACKGROUND OF THE INVENTION 
Field of the Invention 
5 This invention relates to an image processing method 

and apparatus for laying out person images, which have been 
obtained by photographing a plurality of persons, and forming 
a layout image, such as a school photo. This invention also 
relates to a recording medium, on which a program for causing 
10 a computer to execute the image processing method has been 
recorded and from which the computer is capable of reading the 
program* 

Description of t he Pri o r Ar t 
Album prints , typically school photos , are formed 

15 by laying out person images, which have been obtained by 
photographing persons, in predetermined layouts, thus 
obtaining layout images, and printing the layout images. As 
illustrated in Figure 5, if the person images are merely laid 
out in a layout image comprising the person images, positions 

20 and/or sizes of face patterns of the persons will vary for 

different person images, and the layout image will become 
unsightly* Therefore, it may be considered to take photographs 
such that the positions and/or the sizes of the face patterns 
of the persons in the photographs may be uniform* However, in 

25 cases where photographs of a plurality of persons are taken, 



considerable time and labor are required to take the photographs 
by changing the position of the camera in accordance with the 
heights of the persons and/or the sizes of the faces of the 
persons* Also, the position of the camera with respect to the 
persons varies for different cameramen and different dates of 
photographing. Accordingly, it is difficult to take the 
photographs such that the positions and/or the sizes of the 
face patterns of the persons in all of the photographs may be 
uniform. 

Therefore, in cases where a layout image is to be 
formed, when operations for printing from sheets of negative 
film, on which person images have been recorded, are performed, 
the positions of the sheets of negative film and/or the 
positions of photosensitive materials have heretofore been 
adjusted such that the positions and/or the sizes of the face 
patterns of the persons may be uniform among the images. 
Alternatively, a layout image has heretofore been formed by 
laying out person images , which have been obtained by performing 
a trimming process such that the positions and/or the sizes 
of the face patterns of the persons may be uniform. As another 
alternative, a layout image has heretofore been formed by 
printing person images to a comparatively large size, arraying 
the prints of the person images such that the positions of the 
face patterns of the persons may be approximately uniform, and 
again performing the photographing operation. 



However, with the technique for forming a layout 
image by adjusting the positions of the sheets of negative film 
and/or the positions of the photosensitive materials and 
performing the printing operations, and with the technique for 
forming a layout image by arraying the printed images again 
performing the photographing operation, the efficiency of the 
layout image formation and the productivity of the layout images 
cannot be kept high. 

SUMMARY OF THE INVENTION 

The primary object of the present invention is to 
provide an image processing method, wherein a layout image, 
such as a school photo, is capable of being formed efficiently. 

Another object of the present invention is to provide 
an apparatus for carrying out the image processing method. 

The specific object of the present invention is to 
provide a recording medium, on which a program for causing a 
computer to execute the image processing method has been 
recorded and from which the computer is capable of reading the 
program. 

The present invention provides an image processing 
method for obtaining a layout image signal representing a layout 
image, in which a plurality of person images are laid out, from 
a plurality of original image signals, each of the original 
image signals representing a person image, in which a face 
pattern of a person is embedded, the method comprising the steps 



of: 

i) detecting face information from each of the 
original image signals, the face information representing a 
position and/or a size of the face pattern of the person in 

5 the person image represented by each original image signal, 

ii) performing a face pattern normalizing process 
on each of the original image signals and in accordance with 
the detected face information, a plurality of normalized image 
signals being obtained from the face pattern normalizing 

10 process, and 

iii) laying out a plurality of images, which are 
represented by the normalized image signals , in a predetermined 
layout, whereby the layout image signal representing the thus 
formed layout image is obtained. 

15 The term "face information" as used herein means, 

for example, the information representing a displacement 
quantity for transformation of a person image such that the 
position of the face pattern in the person image may coincide 
with a position in a predetermined reference image or a 

2 0 predetermined coordinate position. Also, the term "face 

pattern normalizing process" as used herein means a process 
for transforming the original image signal with an affine 
transformation technique, or the like, such that the positions 
and/or the sizes of the face patterns of the persons may become 

25 uniform among the person images. 



The present invention also provides an image 
processing apparatus for obtaining a layout image signal 
representing a layout image, in which a plurality of person 
images are laid out, from a plurality of original image signals, 
5 each of the original image signals representing a person image, 

in which a face pattern of a person is embedded, the apparatus 
comprising: 

i) detection means for detecting face information 
from each of the original image signals, the face information 

10 representing a position and/or a size of the face pattern of 

the person in the person image represented by each original 
image signal, 

ii) normalization means for performing a face 
pattern normalizing process on each of the original image 

15 signals and in accordance with the detected face information, 

a plurality of normalized image signals being obtained from 
the face pattern normalizing process, and 

iii) editing means for laying out a plurality of 
images, which are represented by the normalized image signals, 

20 in a predetermined layout, and obtaining the layout image signal 

representing the thus formed layout image. 

The present invention further provides a recording 
medium, on which a program for causing a computer to execute 
the image processing method in accordance with the present 

25 invention has been recorded and from which the computer is 



capable of reading the program. 

With the image processing method and apparatus in 
accordance with the present invention, the face patterns of 
the persons in the person images are normalized, and the 
normalized image signals are thereby obtained* The images, 
which are represented by the normalized image signals, are laid 
out in the predetermined layout, and the layout image signal 
representing the thus formed layout image is obtained. 
Therefore, in the layout image reproduced from the layout image 
signal, the positions and/or the sizes of the face patterns 
of the persons become uniform. Accordingly, the layout image, 
in which the positions and/or the sizes of the face patterns 
of the persons are uniform, can be formed efficiently. As a 
result, the productivity of layout images , such as school photos , 
can be enhanced. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram showing an album printing 
system, in which an embodiment of the image processing apparatus 
in accordance with the present invention is employed, 

Figure 2 is an explanatory view showing a person 

image, 

Figure 3 is a flow chart showing how the embodiment 
of the image processing apparatus in accordance with the present 
invention operates , 

Figure 4 is a schematic view showing a layout image, 

6 



in which a plurality of person images have been normalized, 
Figure 5 is a schematic view showing a layout image, 
in which a plurality of person images have not been normalized , 
and 

5 Figure 6 is a schematic view showing layout images 

obtained with a different embodiment of the image processing 
apparatus in accordance with the present invention. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
The present invention will hereinbelow be described 

10 in further detail with reference to the accompanying drawings. 

Figure 1 is a block diagram showing an album printing 
system, in which an embodiment of the image processing apparatus 
in accordance with the present invention is employed. With 
reference to Figure 1, the album printing system comprises 

15 reading means 1 for reading out a plurality of original image 

signals SO, SO, each of which represents a person image. 

The album printing system also comprises detection means 2 for 
detecting face information H from each of the original image 
signals SO, SO, ... having been received from the reading means 

20 1. The face information H represents a displacement quantity, 

which is to be utilized in a face pattern normalizing process 
for uniformizing the positions and/or the sizes of face patterns 
in the person images. The album printing system further 
comprises normalization means 3 for performing affine 

25 transformation on each of the original image signals SO, SO, . . . 



and in accordance with the corresponding face information H, 
and thereby obtaining a plurality of normalized image signals 
SI, SI, • • • representing images, in which the positions and/or 
the sizes of the face patterns of the persons have been 
normalized. The album printing system still further comprises 
editing means 4 for laying out the images, which are represented 
by the normalized image signals SI, SI, . . . , in a predetermined 
layout, and obtaining a layout image signal S2 representing 
a layout image R, in which the plurality of the person images 
have been laid out. The album printing system also comprises 
a printer 5 for reproducing the layout image R from the layout 
image signal S2 . 

The original image signals SO, SO, ... may be obtained 
from a read-out apparatus, such as a scanner, which reads out 
the person images from negative film or positive film. 
Alternatively, the original image signals SO, SO, ... may be 
obtained from image sensing operations performed by a digital 
camera. The thus obtained original image signals SO, SO, ... 
are stored on a storage medium or a storage device. The reading 
means 1 reads the original image signals SO, SO, ... from the 
storage medium or the storage device. 

The detection means 2 comprises edge detecting means 
6 for detecting edges in the face pattern embedded in the person 
image, which is represented by each original image signal SO. 
The detection means 2 also comprises binarization means 7 for 

8 



binarizing the original image signal SO , which has been obtained 
from the edge detection, and obtaining a binary signal SO 1 . 
The detection means 2 further comprises pattern matching means 
8. The pattern matching means 8 performs a pattern matching 
5 process on the binary signal SO ' and a reference signal G, which 

represents a predetermined binary reference image. The 
pattern matching means 8 thus detects at least two points, at 
which the degree of correlation between the person image and 
the reference image is highest. Also, the pattern matching 

10 means 8 determines displacement quantities, which are to be 
utilized for transforming a center position having coordinates 
(x, y) in the face pattern embedded in the person image as 
illustrated in Figure 2 into a center position having 
coordinates (xO, yO) in the reference image, and uniformizing 

15 a size v of the face pattern in the vertical direction and a 

size h of the face pattern in the horizontal direction. The 
detection means 2 feeds information, which represents the thus 
determined displacement quantities, as the face information 
H into the normalization means 3. The reference image 

20 represents a contour image of a face having a predetermined 

size, which contour image is located such that it may have the 
center position at the predetermined coordinates (xO, yO). 

In accordance with the face information H received 
from the detection means 2, the normalization means 3 performs 

25 affine transformation on the original image signal SO and 



obtains a normalized image signal SI. The affine 
transformation is performed with Formula (1) shown below. 




/ a b\ /xl\ /c 
+ 



\ 



7 



(1) 



\ 



in which (xl , yl ) represents the coordinate values of the person 
image before being transformed,, and (x2, y2) represents the 
coordinate values of the person image after being transformed. 

With the coordinate transformation performed with 
Formula (1), all of the image size enlargement or reduction 
of the person image represented by the original image signal 
SO, the rotation of the entire person image, and the parallel 
translation of the person image are achieved simultaneously. 

The coefficients a, b, c, and d in Formula (1) are 
determined in the manner described below. Specifically, 
Formula (1) can be divided into Formulas (2) and (3) shown below. 
x2=axl+byl+c (2) 
y2=-bxl+ayl+d (3) 
The two points in the person image, which are detected by the 
pattern matching process as the points having the highest degree 
of correlation with the reference image, may have coordinates 
(xll, yll) and coordinates (xl2, yl2). Also, the two points 
in the reference image, which correspond to the two points in 
the person image, may have coordinates (x21, y21) and 

10 



coordinates (x22,y22). In such cases, Formulas (4), (5), (6), 
and (7) shown below obtain* 

x21=axll+byll+c (4) 
x22=axl2+byl2+c (5) 
5 y21=-bxll+ayll+d (6) 

y22=-bxl2+ayl2+d ( 7 ) 

The four coefficients a, b, c, and d can be calculated by solving 
Formulas (4), (5), (6), and (7). 

How this embodiment operates will be described 
10 hereinbelow. Figure 3 is a flow chart showing how the 

embodiment of the image processing apparatus in accordance with 
the present invention operates . 

With reference to Figure 3, firstly, in a step SI, 
the reading means 1 reads the original image signals SO, SO, ... 
15 from the storage medium or the storage device. In a step S2, 

each of the original image signals SO, SO, ... is fed into the 
detection means 2. In a step S3, the edge detecting means 6 
of the detection means 2 performs a differentiation process, 
or the like, on the original image signal SO and detects edges 
20 in the person image represented by the original image signal 

SO . In a step S4 , the binarization means 7 performs a binarizing 
process on the original image signal SO, which has been obtained 
from the edge detection, the binary signal SO ' is obtained from 
the binarizing process. Thereafter, in a step S5, the pattern 
25 matching means 8 calculates the displacement quantities, which 

11 



are to be utilized for transforming the center position having 
the coordinates (x, y ) in the face pattern embedded in the person 
image into the center position having the coordinates (xO, yO) 
in the reference image, and uniformizing the size v of the face 
5 pattern in the vertical direction and the size h of the face 

pattern in the horizontal direction. The information, which 
represents the calculated displacement quantities, is fed as 
the face information H into the normalization means 3, 

In a step S6, the normalization means 3 calculates 

10 the affine transformation coefficients with Formula (1) and 

in accordance with the face information H, performs the affine 
transformation on the original image signal SO, and obtains 
the normalized image signal Si representing the image, in which 
the position and/or the size of the face pattern of the person 

15 has been normalized. In a step S7, a judgment is made as to 

whether all of the original image signals SO, SO, ... having 
been read by the reading means 1 have been or have not been 
normalized. In cases where it has been judged that all of the 
original image signals SO, SO, ... have not been normalized, 

20 the processes from the step S2 to the step S6 are iterated. 

In cases where it has been judged that all of the original image 
signals SO, SO, ... have been normalized, in a step S8, the 
editing means 4 lays out the images, which are represented by 
the normalized image signals SI, SI, in a predetermined 

25 layout and obtains the layout image signal S2 representing the 

12 



layout image R. Thereafter, in a step S9, the printer 5 prints 
the layout image R in accordance with the layout image signal 
S2. 

In this manner, with this embodiment, the process 
for normalizing the face patterns of the persons in the person 
images is performed on the original image signals SO, SO, . .. 
representing the person images, and the normalized image 
signals Si, SI, ... are thereby obtained. The images, which 
are represented by the normalized image signals SI, SI, 
are laid out in the predetermined layout, and the layout image 
signal S2 representing the thus formed layout image is obtained. 
Therefore, as illustrated in Figure 4, in the layout image R 
reproduced from the layout image signal S2, the positions and/or 
the sizes of the face patterns of the persons become uniform. 
Accordingly, the layout image R, in which the positions and/or 
the sizes of the face patterns of the persons are uniform, can 
be formed efficiently. As a result, the productivity of layout 
images R, R, such as school photos, can be enhanced. 

In the embodiment described above, the detection 
means 2 detects at least two points having the highest degree 
of correlation by performing the pattern matching process and 
calculates the affine transformation coefficients a, b, c, and 
d to be utilized for the normalization of the position and/or 
the size of the face pattern of the person in the person image. 
In cases where only the position of the face pattern of the 



person in the person image is to be compensated for, only one 
point having the highest correlation may be detected, and the 
position of the face pattern may be uniformized in accordance 
with the detected point. In this manner, the normalization of 
the face pattern can be performed quickly. 

Also, in the embodiment described above, the 
detection means 2 detects the edges in the person image, 
performs the pattern matching with the reference image, and 
thereby calculates the displacement quantities . 
Alternatively, the displacement quantities may be calculated 
with a technique described in, for example, Japanese Unexamined 
Patent Publication No. 6 ( 1994 )-67320 . Specifically, the 
person image may be divided into a plurality of regions in 
accordance with the distributions of the hue and saturation 
values in the person image, and a face candidate region may 
be detected. A face region may then be detected in accordance 
with the shape of a neighboring region, which is located in 
the vicinity of the face candidate region. Thereafter, the 
pattern matching process as described above may be performed 
on the signal representing the face region, and the displacement 
quantities may thereby be calculated. As another alternative, 
an ellipse circumscribed about a simply detected face candidate 
region may be formed, and the displacement quantities with 
respect to the reference image may be calculated by taking the 
center point of the ellipse as the center position of the face 



pattern having the coordinates (x, y) . The face region may be 
detected with one of various other techniques. For example, 
the face region may be detected by utilizing neural network 
techniques described in, for example, Japanese Unexamined 
5 Patent Publication Nos. 5 ( 1 993 ) -274438 and 5 ( 1993 ) -307605 . 

Further, in the aforesaid embodiment, the layout 
image R is formed by normalizing the face patterns of a plurality 
of different persons* Alternatively, as illustrated in Figure 
6, an album may be formed by combining a plurality of layout 
10 images, each of which comprises a plurality of images of the 

same person. In such cases, the image processing apparatus in 
accordance with the present invention can be utilized for 
normalizing the positions and/or the sizes of the face patterns , 
which are embedded in the images constituting each layout image. 
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What is claimed is: 

1 . An image processing method for obtaining a layout 
image signal representing a layout image, in which a plurality 
of person images are laid out, from a plurality of original 
image signals, each of the original image signals representing 
a person image, in which a face pattern of a person is embedded, 
the method comprising the steps of: 

i) detecting face information from each of the 
original image signals, said face information representing a 
position and/or a size of the face pattern of the person in 
the person image represented by each original image signal, 

ii) performing a face pattern normalizing process 
on each of the original image signals and in accordance with 
said detected face information, a plurality of normalized image 
signals being obtained from said face pattern normalizing 
process, and 

iii) laying out a plurality of images, which are 
represented by said normalized image signals, in a 
predetermined layout, whereby the layout image signal 
representing the thus formed layout image is obtained. 

2. A method as defined in Claim 1 wherein said face 
pattern normalizing process is performed by utilizing affine 
transformation • 

3. An image processing apparatus for obtaining a 
layout image signal representing a layout image, in which a 



plurality of person images are laid out, from a plurality of 
original image signals, each of the original image signals 
representing a person image, in which a face pattern of a person 
is embedded, the apparatus comprising: 

i) detection means for detecting face information 
from each of the original image signals, said face information 
representing a position and/or a size of the face pattern of 
the person in the person image represented by each original 
image signal, 

ii) normalization means for performing a face 
pattern normalizing process on each of the original image 
signals and in accordance with said detected face information, 
a plurality of normalized image signals being obtained from 
said face pattern normalizing process, and 

iii) editing means for laying out a plurality of 
images, which are represented by said normalized image signals, 
in a predetermined layout, and obtaining the layout image signal 
representing the thus formed layout image. 

4. An apparatus as defined in Claim 3 wherein said 
face pattern normalizing process is performed by utilizing 
affine transformation, 

5 . A recording medium, on which a program for causing 
a computer to execute an image processing method has been 
recorded and from which the computer is capable of reading the 
program, the image processing method comprising obtaining a 



layout image signal representing a layout image, in which a 
plurality of person images are laid out, from a plurality of 
original image signals, each of the original image signals 
representing a person image, in which a face pattern of a person 
is embedded, 

wherein the program comprises the procedures of: 

i) detecting face information from each of the 
original image signals, said face information representing a 
position and/or a size of the face pattern of the person in 
the person image represented by each original image signal, 

ii) performing a face pattern normalizing process 
on each of the original image signals and in accordance with 
said detected face information, a plurality of normalized image 
signals being obtained from said face pattern normalizing 
process, and 

iii) laying out a plurality of images , which are 
represented by said normalized image signals, in a 
predetermined layout, whereby the layout image signal 
representing the thus formed layout image is obtained. 

6. A recording medium as defined in Claim 5 wherein 
said face pattern normalizing process is performed by utilizing 
affine transformation. 



ABSTRACT OF THE DISCLOSURE 
Face information is detected from each of original 
image signals, each of the original image signals representing 
a person image, in which a face pattern of a person is embedded. 
5 The face information represents a position and/or a size of 

the face pattern of the person in the person image represented 
by each original image signal • A face pattern normalizing 
process is performed on each of the original image signals and 
in accordance with the detected face information, and a 

10 plurality of normalized image signals are obtained from the 

face pattern normalizing process . A plurality of images , which 
are represented by the normalized image signals, are laid out 
in a predetermined layout, and a layout image signal 
representing the thus formed layout image is obtained. A layout 

15 image, such as a school photo, is thus capable of being formed 

efficiently . 
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patentability as defined In Title 37, Code of Federal Regulations, 
Section 1.56. 
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Japanese Language Declaration 

(0*SfM*) 2091-205P 



*JUi. *BSSfe*»3 5»1 1 9*(a)-(d)aXHt3 6 5* 

(b)#teae*na<D, *B4U*©B©#fe< *b&}b 

Prior Foreign Application^) 

( patent) 336998/1998 Japan 
(Number) (Country) 

<H45) 



I hereby claim foreign priority under Title 35, United States Code, 
Section 119 (a)-{d) or 365(b) of any foreign application(s) for patent 
or inventor's certificate, or 365(a) of any PCT International 
application which designated at least one country other than the 
United States, listed below and have also identified below, by 
checking the box, any foreign application for patent or Inventor's 
certificate, or PCT International application having a filing date 
before that of the application on which priority is claimed. 

Priority Not Claimed 
L 

27/11/1998 

(Day/Month/Year Filed) D 



(Number) (Country) 
(**) <H*> 



(Day/Month/Year Filed) 

I hereby claim the benefit under Title 35, United States Code, 
Section 119(e) of any United States provisional application^) listed 
below. 



(Application No.) 
(WB##) 



(Filing Date) 
(ffl»H) 



Tia<D*B»A*3 5IB1 2 0*^|V>TTia(D* 

«A*tt3 6 5 &(c)te£-r fc=fe»L*-*\ * 
It. *Biil©#lll«KH^ft#36s*B*A»3 5«112* 

T*©*W*fcA^3*vfc, *#«R«feft«E3 7»1*5 6:fl 



(Application No.) 
(WIS#*) 



(Application No.) 



(Filing Date) 
(UBH) 

(Filing Date) 
(fcHHB) 



(Application No.) 



(Filing Date) 



! hereby claim the benefit under Title 35, United States Code, 
Section 120 of any United States application^), or 365(c) of any 
PCT International application designating the United States, listed 
below and, insofar as the subject matter of each of the claim* of 
this application Is not disclosed in the prior United States or PCT 
International application in the manner provided by the first 
paragraph of Title 35, United States Code Section 112, I 
acknowledge the duty to disclose information which is material to 
patentability as defined in Title 37, Code of Federal Regulations, 
Section 1.56 which became available between the filing date of the 
prior application and the national or PCT international filing date of 
application. 



(Status: 



Patented, Pending, Abandoned) 



(Status: Patented, Pending, Abandoned) 

mU : WWrPTSr, + . 

I hereby declare that all statements made herein of my own 
knowledge are true and that all statements made on information 
and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or 
imprisonment, or both, under Section 1001 of Title 18 of the 
United States Code and that such willful false statements may 
jeopardize the validity of the application or any patent issued 
thereon. 
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Japanese Language Declaration 



2091-205P 



POWER OF ATTORNEY: As a named inventor, 1 hereby appoint 
the following attorney(s) and/or agent(s) to prosecute this 
application and transact all business in the Patent and Trademark 
Office connected therewith (list name and registration number) 



TERRELL C. BIRCH {Reg. No. 19,382) 
RAYMOND C. STEWART (Reg. No. 21,0661 
JOSEPH A. KOLASCH {Reg. No. 22,4831 
ANTHONY L 8IRCH (Reg. No. 26,122) 



JAMES M. SLATTERY (Reg. No. 28.380) 
BERNARD L SWEENEY (Reg. No. 24,4481 
MICHAEL K. MUTTER {Reg. No. 29,680) 
CHARLES GGRENSTEIN (Reg. No. 29,271) 



GERALD M. MURPHY {Reg. No. 28,977) 
LEONARD R. SVENSSON (Reg. No. 30,330} 
TERRY L. CLARK (Reg. No. 32,644) 
ANDREW D. MEIKLE {Reg. No. 32,868) 



MARC S. WEINER (Reg. No. 32,181) 
ANDREW F. REISH (Reg. No. 33,443) 
JOE M. MUNCY [Reg. No. 32.334) 
C. JOSEPH FARACI (Reg. No. 32,350) 



Send Correspondence to: 



BIRCH, STEWART, KOLASCH & BIRCH, LLP 
P.O. BOX 747 

FALLS CHURCH, VA 22040-0747 
TEL: {703) 205*8000 



^ K&fiSSiS • i& teT5fS#f) Direct Telephone Calls to: (name and telephone number) 



BIRCH, STEWART, KOLASCH & BIRCH, LLP 
TEL: (703) 205-8000 







Full name of sole or first inventor 

Nobuyoshi Naka j ima 






Inventors signature , \ / J c^O Date 

A/^i,A4**fa M November 16, 


mm 




R£si(?ence f ' 

Kaisei-machi , Japan 


mm 




Citizenship 

Japan 






Post Office Address c/o Fuji Photo Film Co . , 

Ltd., 798 Miyanodai, Kaisei-machi, 


Ashigarakami-gun , Kanagawa-ken , 
Japan 






Full name of second joint inventor, if any 




Stir 


Second inventor's signature Date 






Residence 






Citizenship 






Post Office Address 








{Supply similar Information and signature for third and subsequent 
joint Inventors.) 
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